Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message validation optimization #6462

Merged
merged 47 commits into from
Oct 4, 2024

Conversation

cristure
Copy link
Contributor

Reasoning behind the pull request

  • Sometimes a message could be processed multiple times with no real benefit. Thus we need to introduce a mechanism that will check if the message has been previously processed in a set time span and exit early on if that is the case.

Proposed changes

  • Add a map of timeCachers. Before initializing every interceptor we add the timeCacher to the map. When they will try to process the message they will retrieve their cacher from the global map and check if the message has been processed before.

Testing procedure

  • TBD

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

  • was the PR targeted to the correct branch?
  • if this is a larger feature that probably needs more than one PR, is there a feat branch created?
  • if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

epochStart/bootstrap/syncEpochStartMeta.go Outdated Show resolved Hide resolved
factory/processing/processComponents.go Outdated Show resolved Hide resolved
process/interceptors/multiDataInterceptor.go Outdated Show resolved Hide resolved
process/interceptors/multiDataInterceptor.go Show resolved Hide resolved
AdoAdoAdo
AdoAdoAdo previously approved these changes Oct 1, 2024
@sstanculeanu sstanculeanu self-requested a review October 1, 2024 12:07
epochStart/bootstrap/process_test.go Outdated Show resolved Hide resolved
epochStart/bootstrap/syncEpochStartMeta.go Show resolved Hide resolved
node/nodeRunner.go Outdated Show resolved Hide resolved
@@ -321,7 +323,11 @@ func (nr *nodeRunner) executeOneComponentCreationCycle(
}

log.Debug("creating bootstrap components")
managedBootstrapComponents, err := nr.CreateManagedBootstrapComponents(managedStatusCoreComponents, managedCoreComponents, managedCryptoComponents, managedNetworkComponents)
interceptedDataVerifierFactory := factory.NewInterceptedDataVerifierFactory(factory.InterceptedDataVerifierFactoryArgs{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not create it inside process components where it is being used?

the general config is already provided, no changes needed on nodeRunner

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not moved

process/factory/interceptorscontainer/args.go Show resolved Hide resolved
process/interface.go Show resolved Hide resolved
process/mock/interceptedDataVerifierFactoryStub.go Outdated Show resolved Hide resolved
process/mock/interceptedDataVerifierMock.go Outdated Show resolved Hide resolved
update/factory/fullSyncInterceptors.go Outdated Show resolved Hide resolved
update/factory/fullSyncInterceptors.go Show resolved Hide resolved
Co-authored-by: Sorin Stanculeanu <[email protected]>
Comment on lines 999 to 1001
epochStartProvider.interceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
epochStartProvider.interceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMock.InterceptedDataVerifierMock{}, nil
}}
epochStartProvider.interceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 132 to 134
args.InterceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMock.InterceptedDataVerifierMock{}, nil
}}
args.InterceptedDataVerifierFactory = &processMock.InterceptedDataVerifierFactoryMock{}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -62,13 +65,17 @@ func NewEpochStartMetaSyncer(args ArgsNewEpochStartMetaSyncer) (*epochStartMetaS
if check.IfNil(args.MetaBlockProcessor) {
return nil, epochStart.ErrNilMetablockProcessor
}
if check.IfNil(args.InterceptedDataVerifierFactory) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still missing test for this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added.

Comment on lines 270 to 274
args.InterceptedDataVerifierFactory = &processMocks.InterceptedDataVerifierFactoryMock{
CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMocks.InterceptedDataVerifierMock{}, nil
},
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &processMocks.InterceptedDataVerifierFactoryMock{
CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &processMocks.InterceptedDataVerifierMock{}, nil
},
}
args.InterceptedDataVerifierFactory = &processMocks.InterceptedDataVerifierFactoryMock{}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -321,7 +323,11 @@ func (nr *nodeRunner) executeOneComponentCreationCycle(
}

log.Debug("creating bootstrap components")
managedBootstrapComponents, err := nr.CreateManagedBootstrapComponents(managedStatusCoreComponents, managedCoreComponents, managedCryptoComponents, managedNetworkComponents)
interceptedDataVerifierFactory := factory.NewInterceptedDataVerifierFactory(factory.InterceptedDataVerifierFactoryArgs{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not moved

Comment on lines 549 to 551
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Comment on lines 567 to 569
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Comment on lines 602 to 604
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

Comment on lines 656 to 658
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

Comment on lines 500 to 502
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

Comment on lines 569 to 571
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

Comment on lines 607 to 609
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

Comment on lines 660 to 662
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args.InterceptedDataVerifierFactory = &mock.InterceptedDataVerifierFactoryMock{CreateCalled: func(topic string) (process.InterceptedDataVerifier, error) {
return &mock.InterceptedDataVerifierMock{}, nil
}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

@@ -79,6 +81,9 @@ func NewShardInterceptorsContainerFactory(
if check.IfNil(args.PeerSignatureHandler) {
return nil, process.ErrNilPeerSignatureHandler
}
if check.IfNil(args.InterceptedDataVerifierFactory) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

@@ -80,6 +81,9 @@ func NewMetaInterceptorsContainerFactory(
if check.IfNil(args.PeerSignatureHandler) {
return nil, process.ErrNilPeerSignatureHandler
}
if check.IfNil(args.InterceptedDataVerifierFactory) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

ValidInterceptedData interceptedDataStatus = iota
InvalidInterceptedData
validInterceptedData interceptedDataStatus = iota
invalidInterceptedData

interceptedDataStatusBytesSize = 8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe move the last const outside to have clearer the "enum" values for the status

@AdoAdoAdo AdoAdoAdo merged commit c49cde8 into feat/equivalent-messages Oct 4, 2024
5 checks passed
@AdoAdoAdo AdoAdoAdo deleted the message_validation_optimization branch October 4, 2024 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants